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Asymptotic Control for a Class of Piecewise Deterministic Markov 
Processes Associated to Temperate Viruses 

Dan Goreac ^ 


Abstract 

We aim at characterizing the asymptotic behavior of value functions in the control of piece- 
wise deterministic Markov processes (PDMP) of switch type under nonexpansive assumptions. 

For a particular class of processes inspired by temperate viruses, we show that uniform limits of 
discounted problems as the discount decreases to zero and time-averaged problems as the time 
horizon increases to infinity exist and coincide. The arguments allow the limit value to depend 
on initial configuration of the system and do not require dissipative properties on the dynam¬ 
ics. The approach strongly relies on viscosity techniques, linear programming arguments and 
coupling via random measures associated to PDMP. As an intermediate step in our approach, 
we present the approximation of discounted value functions when using piecewise constant (in 
time) open-loop policies. 
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1 Introduction 

We focus on the study of some asymptotic properties in the control of a particular family of piecewise 
deterministic Markov processes (abbreviated PDMP), non diffusive, jump processes introduced in 
the seminal paper [24] . Namely, we are concerned with the existence of a limit of the value functions 
minimizing the Cesaro-type averages of some cost functional as the time increases to infinity for 
controlled switch processes. The main theoretical contribution of the paper is that the arguments 
in our proofs are entirely independent on dissipativity properties of the PDMP and they apply 
under mild nonexpansivity assumptions. Concerning the potential applications, our systems are 
derived from the theory of stochastic gene networks (and, in particular, genetic applets modelling 
temperate viruses). Readers wishing to get acquainted to biological or mathematical aspects in 
these models are referred to [16], |37j, [23], [22], [33]). 

Switch processes can be described by a couple where the first component is 

a pure jump process called mode and taking its values in some finite set M. The couple process 
is governed by a jump rate and a transition measure, both depending on the current state of 
the system. Between consecutive jumps, evolves according to some mode-dependent flow. 

Finally, these characteristics (rate, measure, flow) depend on an external control parameter u. 
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Precise assumptions and construction make the object of Section [21 In connection to these jump 
systems, we consider the Abel-type (resp. Cesaro-type) average 


(70, xo) := inf® [/~ (^ 70 ,- 0 ,-^ 7^7o,xo,- 


I^T (7o,a:o) := inf^E /i (77°’*“’“ 


j^70,xo,n 


, Ut) dt] , 
,ut)dt , 


and investigate the existence of limits as the discount parameter 5^0, respectively the time 
horizon T —>• 00 . 

In the context of sequences of real numbers, the first result connecting asymptotic behavior of 
Abel and Cesaro means goes back to Hardy and Littlewood in [36]. Their result has known several 
generalizations: to uncontrolled deterministic dynamics in [291 XIII.5], to controlled deterministic 
systems in I2|, |3| , etc. 

Ergodic behavior of systems and asymptotic of Cesaro-type averages have made the object of 
several papers dealing with either deterministic or stochastic control systems. The partial differ¬ 
ential system approach originating in [12] relies on coercitivity of the associated Hamiltonian (see 
also |5| for explicit criteria). Although the method generalizes to deterministic (resp. Brownian) 
control systems in [2| (resp. [3]), the main drawback resides in the fact that, due to the ergodic 
setting, the limit is independent of the initial condition of the control system. Another approach 
to the asymptotic behavior relies on estimations on trajectories available under controllability and 
dissipativity assumptions. The reader is referred to for the deterministic setting or 

US], m for Brownian systems. Although the method is different, it presents the same drawback 
as the PDE one: it fails to give general limit value functions that depend on the initial data. 

In the context of piecewise deterministic Markov processes, the infinite-horizon optimal control 
literature is quite extensive (ESI, m, m, DP, m, etc.). To our best knowledge, average control 
problems have first been considered in an impulsive control framework in [T7| and |31] . The first 
papers dealing with long time average costs in the framework of continuous control policies are 
m and [18] (see also IZH). The problem studied in the latter papers is somewhat different, since 
it concerns inf limsupyE 

“ T^oo 1 


T I- /_ ,7o,3;o,« dt , thus leading to an inf/sup formulation. 


while, in our case, we deal with a sup/inf formulation. Moreover, the methods employed are 
substantially different. Our work should be regarded as a complement to the studies developed in 
the cited papers. 

A nonexpansivity condition has been employed in [33] in connection to deterministic control 
systems allowing to obtain the existence of a general (uniform) limit value function. This method 
has been (partially) extended to Brownian control systems in [T3]. In both these papers, convenient 
estimates on the trajectories in finite horizon allow to prove the uniform continuity of Cesaro 
averages Vr and an intuition coming from repeated games theory (inspired by [35]) gives the 
candidate for the limit value function. If the convergence to this limit value function is uniform, 
the results of [33] for deterministic systems yield the equivalence between Abel and Cesaro long-time 
averages. This latter assertion is still valid for controlled Brownian diffusions (see m Theorems 
10 and 13]) and (to some extent) for piecewise deterministic Markov processes (see [321 Theorem 
4.1]). 

In the present paper, we generalize the results of [33] and m to the framework of switch 
piecewise deterministic Markov processes. The methods are based on viscosity solutions arguments. 
We deal with two specific problems. The key point is, as for Brownian systems, a uniform continuity 
of average value functions with respect to the average parameter (J or T). However, the approach 
in m benefits from dynamic programming principles, which, within the framework of PDMP, are 
easier obtained for Abel means (discounted functions v^). This is why, results like [141 Proposition 
7 and Theorem 8] are not directly applicable and we cannot make use of the already mentioned 
intuition on repeated games. To overcome this problem, we proceed as follows : if the system 
admits an invariant compact set, we prove the uniform continuity of and use the results 
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in |32l Theorem 4.1] to show (in Theorem 0]) that this family admits a unique adherent point with 
respect to the topology of continuous functions and, hence, it converges uniformly. This implies 
the existence of lim Vt and the limit is uniform with respect to the initial data. 

T^oo 

The second problem is proving the uniform continuity of under explicit noxexpansivity 

conditions. In the Brownian setting, this follows from estimates on the trajectories and a natural 
coupling with respect to the same Brownian motion in m Lemma 3]. For switch PDMP, we obtain 
similar results (in a convenient setting) by using some reference random measure generated by the 
process. Although the second marginal of this coupling might not come from a controlled PDMP, 
it is shown to belong to a convenient class of measures by using linear programming techniques 
(developed in [33], [3l] and inspired by Krylov [H]). 

Let us now explain how the paper is organized. We begin with fixing some notations employed 
throughout the paper (in Subsection EH). We proceed by recalling the construction of controlled 
PDMP of switch type and present the main assumptions on the characteristics in Subsection 12.21 
In Subsection 12.31 we introduce the concept of invariance with respect to PDMP dynamics, the 
value functions (Cesaro and Abel averages) and the occupation measures associated to controlled 
dynamics (taken from [M])- The main contributions of our paper are stated in Section EJ We begin 
with introducing a very general, yet abstract nonexpansivity condition in Subsection 13.11 Condition 
EJ The first main result of the paper (Theorem Ej) is given in Subsection 13.21 This result states 
that whenever the nonexpansivity Condition [2] is satisfied, there exists a unique limit value function 

lim Vt = limu'^ independent of the average considered (Abel/ Cesaro). In subsection 13.31 we give 
T —^oo (5—^0 

explicit nonexpansive conditions on the dynamics and the cost functional implying Condition [2j 
The second main result of the paper fSubsection 13.41 Theorem [7|) provides an explicit construction 
of (pseudo-)couplings. 

We proceed with a biological framework justifying our models in Section E| We present the 
foundations of Hasty’s model for Phage A inspired by m in Subsection 14.11 In order to give a hint 
to our readers less familiarized with mathematical models in systems biology, we briefly explain 
how a PDMP can be associated to Hasty’s genetic applet in Subsection 14.21 Finally, the aim of 
Subsection 14.31 is to give an extensive choice of characteristics satisfying all the assumptions of the 
main Theorem [3 

Section El gives the proof of the first main result (Theorem Ej). First, we prove that Condition 
[2] implies the equicontinuity of the family of Abel-average value functions ■ Next, we recall 

the results in |32] on Abel-type theorems to conclude. The results of this section work in all the 
generality of |l8] (see also [33j, [34]L 

The proof of the second main result (Theorem [7|) is given in Section [6j The proof is based on 
constructing explicit couplings satisfying Condition [2] and it relies on four steps. The first step 
is showing that the value functions can be suitably approximated by using piecewise constant 
open-loop policies. This is done in Subsection 16.21 The proof combines the approach in m with 
the dynamic programming principles in |35|. We think that neither the result, nor the method are 
surprising but, for reader’s sake, we have provided the key elements in the Appendix. The second 
step is to interpret the system as a stochastic differential equation (SDE) with respect to some 
random measure fSubsection 16.3p . The third step (Subsection 16.4p is to embed the solutions of 
these SDE in a space of measures satisfying a suitable linear constraint via the linear programming 
approach. To conclude, the fourth step (given in Subsection 16.51) provides a constructive (pseudo-) 
coupling using SDE estimates. 


2 Preliminaries 

2.1 Notations 

Throughout the paper, we will use the following notations. 
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Unless stated otherwise, the Euclidian spaces for some > 1 are endowed with the usual 
Euclidian inner product {x, y) = (y*) x, where y* stands for the transposed, row vector and with 
the associated norm |x| = 1 / {x, x), for all x,y £ 

Eor every r > 0 , the set B (0,r) denotes the closed r-radius ball of . 

The set M will denote some finite set. Whenever needed, the set M is endowed with the discrete 
topology. 

Unless stated otherwise, U is a compact metric space referred to as the control space. We let 
^0 (U) denote the space of U-valued Borel measurable functions defined on MxM^ x M_|_. The 
sequence u = {ui,U 2 , ■■■), where E ^0 (U), for all A: > 1 is said to be an admissible control. 
The class of such sequences is denoted by Aad (U) (or simply Aad whenever no confusion is at risk 
concerning U). We introduce, for every n > 1, the spaces of piecewise constant policies 


Aq (U) = E ^0 (U) : u ( 7 , x, t) 
-4"d(U) = {(n™L>iE Ad(U): 


= ( 7 , x) l{o} (A) + E ( 7 , x) l/k Mil (A) 

fc>0 Vn’ n J 

Um E ^0 (U), m > l| . 


As before, we may drop the dependency on U. 

Eor every bounded function ip : MxM'^ x U —> for some A^, A; > 1 which is Lipschitz- 

continuous with respect to the component, we let 


<^max = sup \p^{x,u)\, Lip{p) 

(7,3:,ii)GMxR-^ xU 

Iv^ll = <^max + Lip{p) . 


sup , 

( 7 ,/r,'y,u)sMxR^-^ xU 
xp=y 


\py {x,u) - {y,u)\ 

\x - y\ 


This is the Lipschitz norm of p. 

Whenever K C is a closed set, we denote by C (M x IK; M) the set of continuous real-valued 
functions defined on M x IK. The set BUC (M x IK; M) stands for the family of real-valued bounded, 
uniformly continuous functions defined on M x IK. 

The real-valued function p : — > R is said to be of class Cl if it has continuous, bounded, 

first-order derivatives. The gradient of such functions is denoted by dxP- 

The real-valued function p : MxR^ —)■ R is said to be of class Cl if p ( 7 , •) is of class (7^, for 
all 7 E M. 

Given a generic metric space A, we let B (A) denote the Borel subsets of A. We also let 

V (A) denote the family of probability measures on A. The distance Wi is the usual Wasserstein 
distance on V (A) and Wi^uausdorff is the usual Pompeiu-Hausdorff distance between subsets of 

V (A) constructed with respect to Wi. 

Eor a generic real vector space A, we let cb denote the closed convex hull operator. 


2.2 Construction of Controlled Piecewise Deterministic Processes of Switch 
Type 

Piecewise deterministic Markov processes have been introduced in [23] and extensively studied for 
the last thirty years in connection to various phenomena in biology (see m, m, m, m, issi), 
reliability or storage modelling (in m, [28]), finance (in |3Z]), communication networks ([35]), etc. 
The optimal control of these processes makes the object of several papers (e.g. [25]) [IS]) [20]) etc.). 
Eor reader’s sake we will briefly recall the construction of these processes, the assnmptions, as well 
as the type of controls we are going to employ throughout the paper. 

The switch PDMP is constructed on a space (P, A", P) allowing to consider a sequence of inde¬ 
pendent, [0,1] uniformly distributed random variables (e.g. the Hilbert cube starting from [0,1] 
endowed with its Lebesgue measurable sets and the Lebesgue measure for coordinate, see [26l Sec¬ 
tion 23]). We consider a compact metric space U referred to as the control space. The process is 
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given by a couple ( 7 , X) , where 7 is the discrete mode component and takes its values in some 
finite set M and the state component X takes its values in some Euclidian state space {N > 1). 
The process is governed by a characteristic triple : 

- a family of bounded, uniformly continuous vector fields x U —> such that 

1/^ {x, u) — (y, u)\ < C \x — y\ , for some C > 0 and all x, y G 7 G M and all u G U, 

- a family of bounded, uniformly continuous jump rates x U —> M_|_ such that 

I {x, u) — A^ {y, u)\ <C\x — y\ , for some C > 0 and all x, y G 7 G M and all u G U, 

- a transition measure Q 1 M x x U — > V (MxM^). We assume that this transition measure 
has the particular form 

Q ( 7 , X, U, dedy) = 4+g^(6l,a:,n) {dy) <3° ( 7 , U, d6) , 

for all ( 7 , x,u) G M X x U. The bounded, uniformly continuous jump functions : M x 
X U —> are such that \g^ {6,x,u) — {9,y,u)\ < Clx —y|, for some C > 0 and all 

x,y G M^, all ( 0 , 7 ) G and all tt G U. The transition measure for the mode component is given 
by <5° : M X U —V (M) . For every A C M, the function ( 7 , u) 1 —)■ ( 7 , u, A) is assumed to be 

measurable and, for every ( 7 , tt) G M x U, ( 7 , u, { 7 }) = 0. 

These assumptions are needed in order to guarantee smoothness of value functions in this 
context (see also m for further comments). Of course, more general transition measures Q can 
be considered under the assumptions of [33], [M] and the results of Subsection 13.21 still hold true. 
However, the approach in Section [ 6 ] only holds true for these particular dynamics and it is the 
reason why we have chosen to work under these conditions. Whenever u G Ao (U) and {to, 70 , xq) G 
M+ X MxR'^, we consider the ordinary differential equation 

f ^/70 -io)) t > ta, 

1 = x„. 

Given some sequence u := {ui,U 2 ,---) C AIq (U), the first jump time Ti has a jump rate 
^70 ( 7 o,xo,t)), i.e. P(ri >t) = exp (^-fg \o tii ( 7 o,a;o,s)) ds^ . The 

controlled PDMP is defined by setting (^y'Jo,xo,u^j^^o,xo,u^ _ ^ ^ ^ [0,Ti). The 

post-jump location (Ti,Yi) has Q (jo,xo,t) ,as conditional distribution given 

Ti = r. Starting from (Ti, Yi) at time Ti, we select the inter-jump time T 2 — Ti such that 

P (T 2 - Ti > t / Ti, Ti, Ti) = exp Xr, ^2 (Ti, Ti, s - Ti)) ds^ . 


We set = ^Ti, , if t G [ri,T 2 ). The post-jump location (T 2 ,T 2 ) 

satisfies 

P ((T2, Ta) G / T 2 , Ti, Ti, Ti) = Q (Ti, Ti, Ta - Ti), h) , 

for all Borel set A C MxM-^. And so on. For simplicity purposes, we set (ro,To,To) = (0,7o,xo). 

2.3 Definitions 

Before stating the main assumptions and results of our paper we will need to recall some concepts : 
invariance with respect to PDMP dynamics, the Abel and Cesaro value functions and the embedding 
of trajectories into occupation measures. 
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2.3.1 Invariance 

In order to get convenient estimates on the trajectories, we assume, whenever necessary, that the 
switch system admits some invariant compact set K. For the applications we have in mind, this 
is not a drawback since, for biological systems, we deal either with discrete components or with 
normalized concentrations (hence not exceeding given limits). We recall the notion of invariance. 

Definition 1 The closed set IK is said to be invariant with respect to the controlled PDMP with 
characteristics (/, A, Q) if, for every ( 7 , x) G M x K and every u G Aad, one has G IK, for 

all t > 0, P—a.s. 

Explicit geometric conditions on the coefficients and the normal cone to IK equivalent to the 
property of invariance are given in [331 Theorem 2.8]. Roughly speaking, these properties are derived 
from the sub/superjet formulation of the condition on d^ being a viscosity supersolution of some 
associated Hamilton-Jacobi integrodifferential system. This invariance condition is natural even 
for purely deterministic nonexpansive systems (cf. [S]). It can be avoided either by localization 
procedures or by imposing some relative compactness on reachable sets (or, equivalently, occupation 
measures). We prefer to work under this condition in order to focus on specihc details of our method, 
rather than localization technicalities. 


2.3.2 Value Functions 

We investigate the asymptotic behavior of discounted value functions (also known as Abel-averages) 


v° ( 7 , x) := inf JE 
ueAad 


inf JE 


POO 

Jo 


7 G M, X G , (5 > 0, as the discount parameter J —)• 0 and Cesaro-average values 


Vt ( 7 , x) := inf -E 


h(r]'^'^,X]’^’^,Us)ds 


1 , 


inf -E 


rTnAt 

n>l 


h (rr’“, ^- 1 )) ds 


7 G M, X G t > 0, as the time horizon t -A 00 . The cost function /i : M x x U —> M is 
assumed to be bounded, uniformly continuous and Lipschitz continuous w.r.t. the state component, 
uniformly in control and mode (i.e. \h ( 7 , x,u) — h ( 7 , y,u)\ < C \x — y\ , for some C > 0 and all 
7 G M, X, ?/ G and all u G U). 


2.3.3 The Infinitesimal Generator and Occnpation Measures 

We recall that, for regular functions ip (for example of class G^), the generator of the control process 
is given by 

Cy-ip ( 7 , x) = (/.y (x, u) , ( 7 , x)) P (x, u) / {p ( 6 », x + g^ {9, x, u)) - p ( 7 , x)) ( 7 , u, dO ), 

Jm 

for all ( 7 , x) G MxM^ and any u G Aad (U)- A complete description of the domain of this operator 
can be found, for instance, in [261 Theorem 26.14]. 
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To any (7,x) G MxM-^ and any u G Aad (U), we associate the discounted occupation measure 


( 1 ) 




f 


.-<5t 


u (r: 


y,x,u 
t ’ 


j^'y,x,u 


, Ut) dt 


for all Borel subsets A C MxM^ x U. The set of all discounted occupation measures is denoted by 
eg (7, x) . We also define 

, r ^ G P (MxM^ X U) s.t. V</) : M (M^) , 1 

Links between 0g ( 7 , x) and 0*^ ( 7 jx) will be given in Theorem 0 For further details, the reader is 
referred to [Mj- 


3 Assumptions and Main Results 

In this section, we present the main assumptions and results of our paper. 

We begin with giving an abstract nonexpansivity condition under which the Abel means 

and the Cesaro means (I4)t>o converge uniformly and to a common limit. It is a very general one 

and, in a less general form, it reads ”a coupling fi can be found between a fixed controlled trajectory 

starting from x and another one starting from y such that the difference of costs evaluated on the 

two trajectories be controlled by the distance \x — y\'\ This is essential in proving ergodic behavior. 

In the uncontrolled dissipative case (see, for example m), the couplings are such that the distance 

between the law of Xf and the one of decreases exponentially (is upper-bounded by some term 

\x — y\). In particular, this implies that limu*^ (x) is constant (independent of x). Unlike the 

S —^0 

classical dissipative approach, our framework allows the limit to depend on the initial data. The 
first main result of the paper states that, under the nonexpansivity Condition O the Abel means 
and the Cesaro means iyt)t>o converge uniformly and to a common limit. 

The main drawback of this Condition [2] is that it is abstract (theoretical). In a setting inspired 
by gene networks, we give an explicit condition (Condition E]) on the characteristics of the PDMP 
implying the abstract nonexpansivity condition. The second main result of the paper allows to link 
the explicit Condition E] and the abstract one given before. This is done by constructing suitable 
(pseudo-)couplings and the proof requires several steps. 

3.1 An Abstract Nonexpansivity Condition 

Throughout the section, we assume the following nonexpansivity condition 

Condition 2 For every <5 > 0, every e > 0, every ( 7 , x, y) G M x and every u G Aad (U), 
there exists y, gV ^(M x x such that 

i. xU) = Ge^Q{'y,x)] 

a. y (M X X U, •) G 0^ ( 7 , y) ; 

in. f \h {9, z,w) — h {6', z', w')\ y {dO, dz, dw, d9', dz', dw') < Lip {h) |x — y| -|- e. 
(MxR^ xVf 

Remark 3 Whenever h only depends on the x component (but not on the mode 7 , nor on the 
control u), one can impose 

Wi (^©0 ( 7 , x), 0^ ( 7 , y)^ < |x - y| , 
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where ©o (resp. Q) denote the marginals /i(M,-jU) of measures /i E ©o (resp. ©). One can impose 
the slightly stronger conditions 

Wi (^©0 ( 7 , x) , ©0 ( 7 , y)) <\x-y\ or Wi^Hausdorff ( 7 , x), ©^ ( 7 , < |x - y| 

and the notion of nonexpansivity is transparent in this setting. 

3.2 First Main Result (Existence of Limit Values and Abel-Tauberian Results) 

The first main result of the paper states that, under the nonexpansivity Condition [21 the Abel 
means and the Cesaro means (V)t>o converge uniformly and to a common limit. 

Theorem 4 Let us assume that there exists a compact set K C invariant with respect to 
the piecewise deterministic dynamics. Moreover, we assume Condition [H to hold true for every 
('y,x,y) E M X Then, admits a unique limit u* E C (M x K.;M) and (Vt)t>o converges 

to v* uniformly. 

The proof is postponed to Section (5) In order to prove Theorem (H we proceed as follows. First, 
we prove that Condition [2] implies the equicontinuity of the family of Abel-average value functions 
. Next, we recall the results in [32] on Abel-type theorems to conclude. 

3.3 An Explicit Nonexpansive Framework 

For the remaining of the section, we assume that the control is given by a couple {u,v) E U := C/ x F 
acting as follows : the jump rate (and the measure giving the new mode) only depend on the 
mode component and is controlled by the parameter u. The component X is controlled both by u 
and by v and it behaves as in the general case. One has a vector field / : M x xU xV —> 
a jump rate A : M x x U x V ^ M+ given by 

A-j, {x, u, v) = Xj (u ), 

and the transition measure Q : M x x U x V ^ V (M x R^) having the particular form 

Q (( 7 , x ), u, V, dOdy) = i^y) ( 7 , u, dO ), 

where governs the post-jump position of the mode component. In this case, the extended 
generator of ( 7 , V) has the form 

(3) 

£“’^ 7 >( 7 ,x) = {f^{x,u,v) ,d^ip{'y,x)) + X{-i,u) / (v? (0, x-h (6», x, u, u)) - ( 7 , x)) ( 7 , u, ^0). 

Jm 

We will show that the results on convergence of the discounted value functions hold true under 
the following explicit condition on the dynamics. 

Condition 5 For every 7 E M, every u £U and every x,y E R^, the following holds true 

{f'y{x,u,v) - f.y{y,u,w) ,x -y) , 'I 

sup inf max < s^P\x + g-f {0,x,u,v) - y - g^ {9,y,u,w)\- \x - y\, l<o 

\h{j,x,u,v)-h{'y,y,u,w)\-Lip{h)\x-y\ ) 

Remark 6 1. Whenever h does not depend on the control, the latter condition naturally follows 
from the Lipschitz-continuity of h. 

2. If, moreover, the post-jump position is given by a (state and control free) translation x >->■ 
X -|- y..y (0) , this condition is the usual deterministic nonexpansive one (i.e. 

sup inf {f-^ (x, u, v) - f-^ {y, u,w) ,x - y) < 0 ). 

This kind of jump (up to a slight modification guaranteeing that protein concentrations do not 
become negative) fits the general theory described in f2^. 



3.4 Second Main Result (Explicit Coupling) 

The second main result of the paper allows to link the explicit Condition [5] and the abstract 
Condition [2] in the framework described in Subsection 13.31 

Theorem 7 We assume Condition\^to hold true. Moreover, we assume that there exists a compact 
set K invariant with respect to the PDMP governed by (/, A, Q). Then the conclusion of Theorem\^ 
holds true (i.e. the family admits a unique limit u* E C (M x ]K;M) and (Vt)i>o converges 

to V* uniformly). 

The proof of this result is postponed to Section [6l It is based on constructing explicit couplings 
satisfying Condition [2] and it relies on four steps. The first step is showing that the value functions 
v^ can be suitably approximated by using piecewise constant open-loop policies. The second step 
is to interpret the system as a stochastic differential equation (SDE) with respect to some random 
measure. The third step is to embed the solutions of these SDE in a space of measures satisfying 
a suitable linear constraint via the linear programming approach. To conclude, the fourth step 
provides a constructive (pseudo-) coupling using SDE estimates. 


4 Example of Application 

4.1 Some Considerations on a Biological Model 

We consider the model introduced in m to describe the regulation of gene expression. The model 
is derived from the promoter region of bacteriophage A. The simplification proposed by the authors 
of [37| consists in considering a mutant system in which only two operator sites (known as OR2 
and OR3) are present. The gene cl expresses repressor (Cl), which dimerizes and binds to the 
DNA as a transcription factor in one of the two available sites. The site OR2 leads to enhanced 
transcription, while OR3 represses transcription. Using the notations in [37j, we let Xi stand for 
the repressor, X 2 for the dimer, D for the DNA promoter site, DX 2 for the binding to the OR2 
site, DX 2 for the binding to the OR3 site and DX 2 X 2 for the binding to both sites. We also denote 
by P the RNA polymerase concentration and by n the number of proteins per mRNA transcript. 
The dimerization, binding, transcription and degradation reactions are summarized by 


2 Ai 

K\ (u^v) 

w X2, 

D + X2 

Kiiu) 

w DX2, 

D + X2 

Ksiu) 

^ DX* 2 , 

DX2 + X2 

Kiiu) 

W DX2X2, 

DX2 + P 

^ 4 “^ DX2 + P + nXi 

Xi 

Kaiu,v) 


The capital letters AT*, 1 < i < 4 for the reversible reactions correspond to couples of direct/reverse 
speed functions fc*, fc_j, while Kt and only to direct speed functions kt and k^. Host DNA gyrase 
puts negative supercoils in the circular chromosome, causing A-T-rich regions to unwind and drive 
transcription. This is why, in the model written here, the binding speeds ^2 (to the promoter 
of the lysogenic cycle Prm) ■, ^3 (to OR3), respectively ^4 and the reverse speeds as well as the 
transcription speed kt are assumed to depend only on a control on the host E-coli (denoted by u). 
This control also acts on the prophage and, hence, we find it, together with some additional control 
V, in the dimerization speed ki and the degradation speed 
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4.2 The Associated Mathematical Model 

Let us briefly explain how a mathematical model can be associated to the previous system. 

(A) Discrete/continuous components 

We distinguish between components that are discrete (only affected by jumps) and components 
that also have piecewise continuous dynamics. For the host (E-Coli), we can have the following 
modes: either unoccupied DNA {D = 1, DX 2 = DX 2 = DX 2 X 2 = 0), or binding to OR2 
{D = DX* = DX 2 X 2 = 0, DX 2 = 1), or to OR3 {D = DX* = DX 2 X 2 = 0, DX* = 1) or to 
both sites {D = DX 2 = DX 2 = 0, DX 2 X 2 = !)• It is obvious that these components are discrete 

and they belong to M = jy G {0,1}^ : Ylt=i 7j = 1 1 . Every reaction involving at least one discrete 
component will be of jump-type. We then have the four jump reactions 


K 2 (u) Ki(u) 

D + X2 DX2, D + X2 ^ DX^, 

Kt(u) 

DX2 + X2 ^ DX2X2, DX2 + P DX2 + P + nXi, 

The couple repressor/dimer (Ai, A 2 ) has a different scale and is averaged (has a deterministic 
evolution) between jumps. Hence, we deal with a hybrid model on M x R^. 

(B) Jump mechanism 

Let us take an example. We assume that the current mode is unoccupied DNA (7 = (1,0,0,0)). 
The only jump reactions possible are 

D + X2 DX2 01 D + X2 DX ^. 


The reaction D + X 2 DX 2 means that free DNA will be occupied by one dimer at OR2 

position. Therefore, we have a DNA and a dimer ’’consumed” and an OR2 binding ’’created”. The 
system jumps 

from ( 1 , 0 , 0 , 0 ,xi,X 2 ) to ( 0 , 1 , 0 , 0 ,xi,X 2 — 1 ) • 


Of course, for a consistent mathematical model, since concentrations cannot be negative, to (xi, X 2 ) 
we actually add ( 0 , — min ( 1 , X 2 )). 

The parameter A is chosen as the ’’propensity” function (i.e. the sum of all possible reaction 
speeds) 

"^( 1 , 0 , 0 , 0 ) (^) '^) = -^{ 1 , 0 , 0 , 0 ) (^) = ^2 {u) + (u ), 


The probability for the reaction D + X 2 
speed (i.e. ' 

To summarize, one constructs 


k2{u) 


DX 2 to take place is proportional to its reaction 


Qe((l,O,O,O),(u,u),d0) 

= (( 1 , 0 , 0 , 0 ), u, de) = ’ 

Q ((1,0,0,0) , (xi, X2), U, dOdy) = 6(^X1,x2)+g-,i,e,(xi,x2),u,v) (dy) < 3 ° (( 1 , 0 , 0 , 0 ) , u, d6) , where 
, 5(1,0,0,0) (d, (a^i, 3:2) ,u,v) = ( 0 , - min ( 1 , X2)) l0e{(o,i,o,o),(o,o,i,o)}- 


Remark 8 A special part is played hy the transcription reaction 

DX 2 + P ^4“^ DX 2 + P + nXi 


which is a slow reaction. Details on a possible construction will be given in Subsection\4.3. 


(C) Deterministic flow 
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The deterministic behavior is governed by the (three) reactions 


Ki{u,v) 

2Xi ^ X 2 cind 


Kd{u,v) 


For example, the reaction 2Xi ^ X 2 needs two molecules of reactant Xi and has ki (u, v) 
as speed and produces one X 2 . Then, following the approach in [37], its contribution to the vector 
field is proportional to the product of reactants at the power molecules needed (i.e. xf) and the 
speed ki {u,v). We get —2xfA:i {u,v) for the reactant and +lx\ki {u,v) for the product. Adding 
the three contributions, we get 

f.y ((xi, X 2 ) ,u,v) = {-2ki {u, v) xl + 2k-i {u, v) X 2 - kd {u, v) xi,ki (u, v) xf - k-i {u, v) X 2 ) ■ 


For further details on constructions related to systems of chemical reactions in gene networks, the 
reader is referred to |23|, [ 22 ], |33j, etc. 


4.3 A Toy Nondissipative Model 

Let us exhibit a simple choice of coefficients in the study of phage A. We consider U = V = \Q,1] 
(worst conditions, best conditions for chemical reactions). For the reaction speed, we take 

ki (u) = ki{u + uq) , for i G {±2, ±3, ±4, t} for the jump reactions determined by host, 

ki (u, v) = —uv, k 2 (u, v) = uv, kd (u, v) = uv to reflect a certain competition, 
a 

for all u,v G [0, 1 ] . Here, uq > 0 corresponds to the slowest reaction speed, ki > 0 are real constants 
and a is some maximal concentration level for repressor and dimer. We assume P to toggle between 
0 and 1 and, as soon as P toggles to 1 a transcription burst takes place. To take into account the 
slow aspect of the transcription reaction (see [3ZI), we actually take 


= {( 1 , 0 , 0 , 0 , 0 ) , ( 0 , 1 , 0 , 0 , 0 ) , ( 0 , 1 , 0 , 0 , 1 ) , ( 0 , 0 , 1 , 0 , 0 ) , ( 0 , 0 , 0 , 1 , 0 )} 


This means that binding to OR2 corresponds to two states (0,1,0,0,0) (allowing transcription), (0,1,0,0,1) 
(when transcription has just taken place and is no longer allowed). We set 


A..y (u, v) = A..y {u) = < 


<3° (7,w) = 


{k2 + k^) (n + Uq) 
{k-2 + ^4 + h) {u + Uq) 
(k-2 + ^ 4 ) (u + Uq) 
k -3 (u + Uq) 
k-4 {u + Mo) 


if 7 = ( 1 , 0 , 0 , 0 , 0 ), 
if 7 = ( 0 , 1 , 0 , 0 , 0 ), 
if 7 = ( 0 , 1 , 0 , 0 , 1 ), 
if 7 = ( 0 , 0 , 1 , 0 , 0 ), 
if 7 = ( 0 , 0 , 0 , 1 , 0 ). 


^ ^ + Vo, 1 , 0 , 0 ), if 7 = ( 1 , 0 , 0 , 0 , 0 ) , 

k_2+k4+kt <^( 1 , 0 , 0 , 0 , 0 ) + k.2+L+kt '^( 0 , 0 , 0 , 1 , 0 ) + kXk+kt <^( 0 , 1 , 0 , 0 , 1 ), if 7 = (0,1,0,0,0) , 

0 , 0 , 0 ) + fcif7^Vo,o,i,o), if 7 = (0,1,0,0,1) , 

<^( 1 , 0 , 0 , 0 , 0 ), if 7 = ( 0 , 0 , 1 , 0 , 0 ) , 

^( 0 , 1 , 0 , 0 , 0 ), if 7 = ( 0 , 0 , 0 , 1 , 0 ) . 
if 7 = ( 1 , 0 , 0 , 0 , 0 ), 

if 7 = ( 0 , 1 , 0 , 0 , 75 ), 75 G { 0 , 1 }, 0 = ( 1 , 0 , 0 , 0 , 0 ), 
if 7 = ( 0 , 1 , 0 , 0 , 75 ), 75 G { 0 , 1 }, 0 = ( 0 , 0 , 0 , 1 , 0 ), 
if 7 = ( 0 , 1 , 0 , 0 , 0 ), 0 = ( 0 , 1 , 0 , 0 , 1 ), 

if 7 = ( 0 , 0 , 1 , 0 , 0 ), 0 = ( 1 , 0 , 0 , 0 , 0 ), 

if 7 = ( 0 , 0 , 0 , 1 , 0 ), 

{{XI,X2) ,U,V) = (/.^ {{X1,X2) ,U,V) ,f^ {{XI,X2) ,U,V)) 

= ( —+ 2uVX 2 — UVXl, ^xf — UVX 2 ) ■ 


gj (0, (xi,X 2 )) = < 


' (0,- 

min(l,X 2 )) 

(0, min 

(1, 

OL - X 2 )) 

(0,- 

min(l,X 2 )) 

(min (n 

a 

- a;i) ,0) 

(0, min 

(1, 

a - X 2 )) 

(0, min 

(1, 

a - X 2 )) 


11 














The reader is invited to note that Mx [0, is invariant with respect to the previous dynamics. 
One can either note that for xi,X 2 G [0,a] 

2 

f}. {{0,X2),u,v) = 2x2UV > 0, ((xi,0) = —uv > 0, 

((a, X 2 ) ,u,v) = {2x2 — 3a) uv < 0 , ((xi, a) ,u,v) = uv < 0 , 

or, alternatively, compute the normal cone to the frontier of [0, a]^ (see [331 Theorem 2.8]) to deduce 
that [0, a]^ is invariant with respect to the deterministic dynamics. Moreover, the definition of 
guarantees that {Xi,X 2 ) does not leave [0,a]^. Let us also note that, whenever x = (xi,X 2 ) G 
[0,a]^ , y = {yi,y 2 ) G [0,a]^ , 


(A {x, u, v) - {y, u,v),x-y) 


n 

= —uv 


- (xi +yi) + l 
a 


( 2:1 -yif - 


2 + -{xi +yi) 
a 


(xi - yi) (x 2 - 2/2) + (x 2 - 2/2)^ < 0 . 


This is a simple consequence of the fact that 


1 , 

2 

'2 , , ■ 


1 , 

2 + -{xi + yi) 
a 

-4 

- {xi + yi) + l 
a 

— 

2 -(xi +yi) 

a 


We deduce that 

sup inf (fj (x, u, v) - {y, u,w) ,x - y) < 0. 

Moreover, for all {x,y) G [0, a]^ , 


sup inf (/^ (x, 0 , v) - {y, 0,w) ,x -y) =0. 

It follows that one is not able to find any positive constant c > 0 such that 
sup inf (/^ (x, 0, v) - {y, 0,w) ,x - y) < -c\x - yf , 

for all {x,y) G [0,a]^ and we deal with a non-dissipative system (unlike, for instance, [9]). 

Finally, the function t 1 —?■ t — min(l,t), t 1 —)• t + min(A:,a — t) are Lipschitz continuous with 
Lipschitz constant 1 on [0, a] for all A; > 0. It follows that sup |x + {9, x) — y — g-y {9, y)\ < \x — y\ . 

9&A 


5 Proof of the First Main Result (Theorem [4j) 

In order to prove Theorem |T1 we proceed as follows. First, we recall the link between the set of 
occupation measures 0 q ( 7 , x) and the family 0*^ ( 7 , x) (taken from [33]). Next, we prove that Con¬ 
dition [2] implies the equicontinuity of the family of Abel-average value functions ■ Finally, 

we recall the results in [32] on Abel-type theorems to conclude. 

5.1 Step 1 : Equicontinuity of Abel-average Values 

The following result corresponds to |34l Theorem 7 and Corollary 8 ] for this (less general) setting. 
It gives the link between the set of occupation measures 0 q ( 7 , x ) and the family 0”^ {'y,x). 

Theorem 9 i) For every x G and every 5 > 0, 

x‘^( 7 , x)= inf / h{9,y,u) fi{d9,dy,du). 

JmxR^xU 

a) For every ( 7 , x) G MxM^ and every d > 0, 0*^ ( 7 , x) = co (0 q ( 7 , x)). 
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We are now able to prove the following equicontinuity result. 

Proposition 10 We assume Condition\^to hold true. Then, for every h > 0 and every {'y,x,y) E 
M X one has 


vHlw) -v^ {l,y) <Lip{h)\x-y\ 


Proof. Let us fix 5 > 0 and ( 7 , x, y) E M x R^^. We only need to prove that, for every e > 0, 

( 7 , y) < v^ ( 7 , x) + Lip [h) \x -y\+ e. 

By definition of , for a fixed e > 0, there exists some u E Aad (U) such that 


v^ ( 7 , x) + - > (iE 




IxR^ xU 


h{e,z,w) {de,dz,dw) 


If p is the coupling measure given by Condition [2] and associated to |, we deduce, using Theorem 
[Hi that 


{'y,y) < [ h (^ 9 ', z', w') p [d9, dz, dw, dO', dz', dw') 

JiMxR^xVf 

< / h{9,z,w) pt. xui(^^^dz,duj) + Lip{h)\x— 

JMxR^xU 

< (7, x) + Lip (h) |x — y| + s. 

The proof of our Proposition follows by recalling that e > 0 is arbitrary. ■ 



5.2 Step 2 : Proof of Theorem |4] 

The previous results on existence of a continuity modulus uniform with respect to the discount 
parameter <5 > 0 allows us to prove the existence of a limit value function as <5 —>• 0. Before going 
to the proof of Theorem U we recall the following. 

Lemma 11 (i) }32L Step 1 of Theorem f.l] Let us assume that is a relatively compact 

subset of C (MxR^;R). Then, for every u E C (MxR^;R), every sequence {dm)m>i 
limm^oo<5m = 0 and converges uniformly to v on MxR^ and every e > 0, there exists 

T > 0 such that 

Vt ( 7 , x) > v ( 7 , x) — e, for all ( 7 , x) E MxR'^ and all t >T. 

(ii) }32i Theorem 4-1] Let us assume that [v^)is a relatively compact subset of C'(MxR'^;R). 
Then, for every v £ C (MxR^;R) and every sequence {dm)jn>i such that lirnm^oodm = 0 and 
(converges uniformly to v on MxR^, the following equality holds true 

liminf sup | Vj ( 7 , x) — u ( 7 , x)| = 0 . 

7eM, xeR^ 

(Hi) \32\. Remark 4-2] Let us assume that converges uniformly to some v* £ C (MxR^; R) 

as (5 —)• 0. Then the functions (Vt)^>o converge uniformly on MxR-^ to v* 

lim sup 114 ( 7 , x) — u* ( 7 , x)| = 0 . 
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Remark 12 In the proof of I32i Theorem 4‘iJ} one gives the eondition (i) in Step 1. However, 
in all generality, the eonverse is only partial. Indeed, Step 2 (see f32i Page 174, Eq. (10)] reads 
; For every e > 0, there exists mo > 1 sueh that 

Vg-i (7, x)<v (7, x) + e, 

for all m > mo and all (7,x) G MxM^. Thus, the index t of the subfamily {Vt)^ eonverging to v 
depends directly on the sequence {dm)m>i ■ course, whenever the limit v is independent on the 
choice of 5, so is t and one gets (Hi). 

We are now able to complete the proof of Theorem 01 
Proof of Theorem 01 Let us denote by 

( 4 ) V* (7, x) := lim sup (7, x), 

for every (7, x) G M x IK (the pointwise lim sup). We fix, for the time being, some (j,x) G M x K. 
Then, there exists some sequence such that lim 6m = 0 and lim (7,x) = v* (7,x). 

— 771—^00 m—>-oo 

Due to Condition[ 2 l the sequence is equicontinuous (see Proposition flO)) and, by definition, 

it is also bounded. Then, using Arzela-Ascoli Theorem, it follows that some subsequence (still 
denoted converges uniformly on M x K to some limit function u G C (M x ]K;M) . In 

particular, w (7, x) = v* (7, x) . Using Lemma fTTl (i), one gets that 

( 5 ) lim infV) (7, x) > v* (7, x). 

t—^OO 

Obviously, this argument can be repeated for every (7, x) G M x IK. Let us now consider w G 
C (M X ]K;M) to be an adherence point of the relatively compact family Then, using 

LemmalUKii), one establishes the existence of some increasing sequence itn)n>i such that lim tn = 

— n—7oo 

00 and 

lim sup (7, x) - tc (7,x)| = 0. 

7 eM,a;eR^ 

In particular, it follows that 

(6) rc ( 7 , x) > lim infU) ( 7 , a;), 

for all (7, x) G M X IK. Combining dl]),© and ([ 61 ), one deduces that the unique adherence point of 
(x^)^>o is V*. The convergence of (V))^>o follows by invoking Lemma fTTl fiiiL Our Theorem is now 
complete. ■ 

6 Proof of the Second Main Result (Theorem [7j) 

The proof of Theorem [ 7 ] is constructive and relies on four steps. We begin with recalling the 
Hamilton-Jacobi integrodifferential systems satisfied by the Abel-average functions and Krylov’s 
shaking the coefficient method. The first step is showing that the value functions can be suitably 
approximated by using piecewise constant open-loop policies. The proof in this part strongly rely 
on the tools presented before. The second step is to interpret the system as a stochastic differential 
equation (SDE) with respect to some random measure. The third step is to embed the solutions of 
these SDE in a space of measures satisfying a suitable linear constraint via the linear programming 
approach. To conclude, the fourth step provides a constructive (pseudo-) coupling using SDE 
estimates. 
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6.1 Krylov’s Shaking the Coefficients 

For every <5 > 0, the value function is known to be the unique bounded, uniformly continuous 
viscosity solution of the Hamilton-Jacobi integro-differential system 

(7) 6v^ ( 7 , x) + H (^ 7 , X, ( 7 , x), = 0, 

where the Hamiltonian is defined by setting 

H {'r,x,p,ip) 

:=sup -h{j,x,u)-{f^{x,u),p)-\^{x,u) {p {e,x + {6, x,u)) - p {'j, x)) Q° {'y,u,de) , 

ueu L Jm 

for all G and all bounded function <y 9 : M x ^ M. For further details on the subject, 

the reader is referred to |48j . 

Although uniformly continuous, the value functions are, in general, not of class C^. However, 
adapting the method introduced in m (see also I7]),u^ can be seen as the supremum over regular 
subsolutions of the system ([7]). Alternatively, one can give a variational formulation of with 
respect to an explicit set of constraints. We recall the following basic elements taken from |34j . 

We begin by perturbing the coefficients and consider an extended characteristic triple 

- X UxH (0,1) —> M^, (x, (^x + € U, U2 G H (0,1), 7 € M, 

- A..y : X UxH (0,1) —>■ Ay (x, u^,u^) = Ay (x + u^, , ui G U, U2 G H (0,1) , 7 G M, 

- Q : X UxH ( 0 , 1) —> V (M^) , Q (7, x, A) = Q {^,x + , A + (O, u^)) , where 

A + (0, u^) = {(oi, 02 + V?) : (oi, 02) £ A} , for all x G M^, & U, u'^ G B ( 0 , 1 ) and all Borel set 

A C MxM^. 

One can easily construct the process withtt= [u^,u^) G Aad (Uxi?(0,1)). 

The initial process associated to (/, A, Q) can be obtained by imposing = 0. Let us note that, 
with this construction, 

Q ( 7 ? X, zx ^ 'Ll , dOd'y^ ^x-\-g'y(o,x-\-u‘^,u^) (^ 1 /) Q (t? ^ ? dO'^ . 


6.2 Step 1: Piecewise Constant Open-loop Policies 

The aim of this subsection is to show that the value functions can be approximated by functions 
in which the control processes are piecewise (in time) constant. For Brownian diffusions, this type 
of result has been proven in [40]. In this section we adapt the method of m to our setting by 
hinting to the modifications whenever necessary. Following m, for all n > 1 , we introduce the 
value function 

( 7 , x) = inf JE 

for all ( 7 , x) G MxR-^. 

The main result of the subsection is the following. 




hr 


~.'y,x,u,0 j^'y,x,u,0 


Ut ) dt 


Theorem 13 Let us assume that there exists a compact, convex set K ivhich is invariant with 
respect to the controlled PDMP with characteristics {f,A,Q). Then, for every J > 0, the value 
functions converge uniformly to v^ as the discretization step n increases to infinity 


lim sup 

n^oo^gM, xeK 


X^( 7 ,x) 


— V 


6,n 


{l,x] 


= 0 . 
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The proof relies on the same arguments as those developed in |3D] combined with dynamic 
programming principles. Let us briefly explain the approach. For every n > 1, one begins by 
proving a dynamic programming principle for and involving T A Ti as intermediate time, for 
T G n“^N. The arguments are essentially the same as those in [l 8 j and we only specify when the 
structure of intervenes. Next, one takes a sequence of smooth functions converging 

uniformly to by adapting to the present framework Krylov’s shaking of coefficients method 
introduced in [2] (see also [7] or [33] for the PDMP case). Then, one proceeds by writing the 
Hamilton-Jacobi integrodifferential system satisfied by This equation is e—close to the one 

satisfied by (with a uniform behavior w.r.t. n > 1). Our assertion follows by integrating this 
subsolution condition with respect to the law of the piecewise deterministic Markov process then 
allowing e ^ 0. For our reader’s convenience, we have indicated the main modifications and 
arguments in the Appendix. 

Remark 14 If the invariance condition holds true, then, by applying this result, one only needs 
to check that the nonexpansive Condition [H holds true for all u G (U) for all n large enough 

(larger than some Us). 


6.3 Step 2 : Associated Random Measures and Stochastic Differential Equa¬ 
tions 

Let us fix 7 o G M, xq G and {u, v) G Aad {U x V). The following construction is quite standard 
and makes the object of |26[ Section 26] for more general PDMP (without control) and [261 Section 

41] (when control is present). We let S'o = To = 0 , Sn = Tn — Tn-i, for all n > 1 and = 

. We look at the process ( 7 , A) under (which depends on both 

the initial state ( 70 , 3 ^ 0 ) and the control couple {u,v), but, having hxed these elements and for 

notation purposes, this dependency will be dropped). By abuse of notation, we let 


u, := 


:= Ui (70, Xo, s) lo<s<Ti + ^ Un+l s _ Tn) 1t„<s<T, 


n +1 ’ 


n>l 


(and similar for v). We denote by ¥ the filtration := a : r G [0,t]})^^g . 

The predictable u-algebra will be denoted by V and the progressive fi-algebra by Prog. For the 
general structure of predictable processes, the reader is referred to jTU] Section 26], [33 Proposition 
4.2.1] or m Appendix A2, Theorem T34]. In particular, due to the previous notations, it follows 
that u and v are predictable. 

As usual, we introduce the random measure p on D x (0, 00 ) x M x by setting 

p (w. A) ^ ^ 1 / Xf„(ui) fo^ all uj G D, A G B (0, 00 ) x B (M x M ) . 

fc>i ^ A 'J’fe ’ J J 

The compensator of p is 

3{dsdyde) = A idy)Q^ ( 7 j°’^°’“’^40) ds. 

^ s — 

and the compensated martingale measure (see [26l Proposition 26.7]) is given by g := p — p. 

By construction, for our model, on [T„_i, T„), is a deterministic function of , 

• - ^n-i) and Vn . - r„_i) . in this partic¬ 

ular framework, 

V -‘-n — l -‘n —1 / V ^n — 1 -‘-n—l J ^n — 1 


X, 


■yo,xo,u,v _ ^ 


Tn-Tn 


(^,-f0,X0,U,V ( ')0,X0,U,V y'yo,Xo,U,V rp rp\ f ■yo,Xo,U,V Y'yo,XQ,U,V rp rp W 

+ 5^70,(^7;r^ , Un (^7t„_i > ,ln- In-lj ,Vn (^7r„_i > “ Jn-1 ) ) , 
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hence being a a deterministic function of ■ It follows that the filtration F 

is actually generated by the marked point process (T^., 7 !°’*°’“’^) . As a consequence, Vn+i = 

Vn+i 7 ^°’*°’“’’', •) is a deterministic function of Ti,T^, 7 ^°’*°’“’’^, 7 ^°’^°’“’’' still de¬ 
noted by Vn+i (^Ti, r„, ..., 7 ^°’^°’“’’', . In the case when {u, v) G {U x V) for some 

m > 1 are piecewise constant, Vn+i is of type 




fc >0 


■yo,xo,u,v 'yo,xo,u,v\ ^ /.x 

’ ) If A Mil [t) ■ 

/ \ m ’ m J 


Similar assertion hold true for u. 

We now define the random measure p on H x (0, 00 ) x M by setting 

p {u!, A) =p (a;, A x M'^) , for all a; € H, A ^ B (0, 00 ) x B 
The properties of p imply that the compensator of p is 

p{dsd9) = X ( 7 j°’'^°’“’'^,u<j) , Ug, d9) ds 

and 

q {dsd9) = p {dsd9) — A I'^s) ,Us, d9) ds 

is its martingale measure. Following the general theory of integration with respect to random 
measures (see, for example [38]), the second state component can be identified with the unique 
solution of the stochastic differential equation (SDE) 

/ dX^°’^°’'^’^ = f ~io,xQ,u,v ,ut,vt) dt + ( 9 , , ut, vt) p (dtd9) ,f > 0, 


j^yo,xo,u,v ^ ^ J*p _ 


6.4 Step 3 : Measure Embedding of Solutions 

More general, whenever w is an F-predictable process, we can consider the equation 

r ^ dt + {9, ,ut,Wt) p {dtd9) , t > 0, 

I = yo, p - a.S. 

The assumptions on the coefficients / and g guarantee that, for every po € and every predictable, 
F—valued process w, this equation admits a unique solution Y^^’'^. We fix 5 > 0 and consider some 
(arbitrary) regular test function 0 G (MxM^;M) . Ito’s formula (see [381 Chapter II, Theorem 
5.1]) applied to ( 7 ?'°’^°’“’’^, T^°’^) on [0,T] yields 


[(j) ( 7 ^ 
= H ( 70 , yo) 

/■T 

-fe ' 




j 

Jo 


-H (7t, Yt) + (/7t (F, Ut, Wt) , d^cj) (7t, Yt)) 


Xp—St j \ ^ \ J'yt \^Ti '^zj 1 '^x'r \ iz^ ^Tj/ III 

LJO V (7t> Ut) {(j) (9, Yt + gj, {9, Yt, ut, wt)) - 4> {lt,Yt)) ( 7 *, ut, d9) 


where we have denoted by {'jt,Yt) = y^i^O’"'). By letting T ^ 00 , it follows that the 

occupation measure g V (MxM^ x U x V) given by 


pVO’^ (4) = E 
satisfies 


■ poo 

/ 5e-^AAhr"°’'''\Yy°'^,ut,wt) 

Jo 


dt 


, for 4 G 5 


N 


UxV) 


/MxK^xU 


(£“’’'(/> {9, y) + 6 ((/>( 7 , x) -4i (0, y))) p {d9, dy, du, dv) = 0. 
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We recall that is the generator given by Q- 

There is no reason for the couple to be associated to aU x W valued piecewise 

open-loop control couple. Nevertheless, the previous arguments show that the occupation measure 
^yo,w belongs to 0*^ (tojI/o) (see ([2])). 

Remark 15 Let us note that if there exists a set IK invariant with respect to the PDMP driven by 
{f,X,Q ), then, for all 70 G M, yo £ the occupation measures /i G 0 q ( 70 , 2 / 0 ) satisfy the support 
condition /U (M x KxU xV) = 1. Then, by Theorem\^ the same holds true for ( 70 , 2 / 0 ) o,nd, 
hence, takes its values in IK. Alternatively, one can use Theorem 2.8 (ii)]. 


6.5 Step 4 : Coupling via the Random Measure 

As in the previous arguments, one can define a measure fi gV ^(M X X U X hy setting 

O 


uo 


iu (A X R) = E 
whenever A G B ^(M x x [/ x C)^^ . It is clear that 


= E 


h {6, z,w) — h (O', z', w') I /r (^dO, dz, dw, dO', dz', dw') 

poo 

/ \h ^ Yy°’'",ut,wt)\dt 

Jo 


IxK^xU) 
r-oo 


n{Ax {MxR^ xU xV)) = e ©0 (To, a^o,) and /r ((M x x [/ x C) x R) = G 

0*^ ( 70 , 2 / 0 ), where given in the previous arguments. Convenient estimates for this integral 

term imply the condition ([2]) and, hence, the results on existence of a limit value function. In 
fact (see Remark I14p . in order to prove Theorem [71 it suffices to provide good estimates when the 
process is constructed with piecewise constant (in time) policies (n, v) G (U x V). This is done 
by the following. 


Lemma 16 We assume Condition\^to hold true. Moreover, we assume that there exists a compact 
set E invariant with respect to the PDMP governed by (/, A, Q). Then, there exists u : M+ —)• M+ 
such that lim^^QUJ (e) = 0 and, for every n > 1 and every (n, v) G (U x V), there exists w 
predictable with respect to the filtration such that 


E 


[/ 


6e '^*|h(77 


) 


\ Ut, Vt) - h (^r.^^O,xo,u,v ^yyo,w ^ 


<uj{n 


Proof. Step 0. 

Let us define a set-valued function 


M. xK^ X U X V 3 {'y,x,y,u,v) E (7, x, y, u, v) 

w gV : ^0 gM, 

) {f-({x,u,v) - f.y{y,u,w) ,x-y) <0, ^ 

\x + gy{e,x,u,v)-y - gy{e,y,u,w)\ <\x - y\, 

, |h(7,x,u,u) -/i(7,2/,?x,u;)| < Lip(/i) |x - 2/1 , 

One easily checks that the function has compact values and is upper semicontinuous. Hence, there 
exists some measurable selection 


w 


X X U X V —> V, w (7, x, y, u,v) gE (7, x, y, u, v) , 
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for all ( 7 , X, y, M, f ) E M X X 17 x 17. For further details, the reader is referred to [BJ Subsection 
9.2], 

We construct an F-predictable 17-valued control process w as follows. We begin by fixing T > 0 
(depending on n) and m > 1 (depending on n). The choice of T and m will be made explicit later 
on. Moreover, we assume that K. gB (0, fco), for some fcg > 0. 

Step 1. We consider 

u;b0 := y} xq, yo, ui (0), vi (0)) = s > 0, 

where we have denoted, by abuse of notation. 


ui (s) = ui ( 70 , xo, s), vi {s) = vi ( 70 , xo, s). 

We recall that if s < 7 , one has ui (s) = ui (0,7o,xo) and similar assertions hold true for vi. By 
recalling that IK is invariant with respect to the controlled piecewise deterministic dynamics, one 
gets 


< (/70 { xq , Us , Vs ) - /70 ,a ^0 - 2/0) +C (^|xo - ^0,xo,«,i;;7o| 

( 8 ) < c (|xo - + |y - 


_ ^ 0 , 3 /o,«,'U' ’ ,70 

^ s 


for all 0 < s < ;i. Similarly, 
(9) 


i. 

n — 




:. 0 ,a;o,n,'y; 7 o ’ ; 7 o 


^u,xo,u,«,70 _ 


— Lip (h) 

^0,xo,u,v,'y(i _ ^^0,yo,M,«jl’°;7o 


for all 0 < s < ^. The constant c > 1 is generic, independent of 5 > 0, n, s, x, y, u and is allowed to 
change from one line to another. We dehne the control process by setting 


— n 


+ w 


^^0,yo,u,w^’Ono 





Then the estimates in Q hold true for s E [O, if substituting 

1,2 1,11 1^1 /F.O,3;o,u,t';7o ^0,VO,u,w^’^\'yo 

W^’ = W^’ +W [jo,^2 ,^2 ,Ui 

\ n n 


to We set 



and so on, to define ..., We fix some G V and let -t- 

t(;°ls>r, where [■] denotes the floor function. As consequence, by recalling that, prior to Ti, both 
j^'yo,xo,u,v yj/o.-u' aj.g deterministic and can be identified with (resp. ’ ’'>' 0 )^ 

we use dSI) to get 


( 10 ) 


j^'yo,xo,u,v _ yVo^^ 


i2 Cf 

< ko - 2/0 + —, 
n 


for all t < T on t < Ti. It follows that 


( 11 ) 


j^ 70 ,xo,u,tJ 


yVO^w^ 


< ko - 2 / 0 I + 



t 
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or all i < T on t <Ti. Moreover, on Ti < T, using (Pii) and (fTU|l and recalling that IK C -B (0, ko) 
is invariant (see also Remark [15]) one has 


Y'yo, 3 :o,u,v _ 


-d> 


*0,a;o,n,'u;7o 

0,yo,u,w''^’°-,-yo 

Ti- 


570 1, UT,, 


< 


^0,a:o,n,'u;7o _ ^0,yo,u,w^’°-,-yo 


C 

H— 
n 


< min 


cTi 


3^0 - 5 o| ^-,2feo + 


n 


n 


( 12 ) 


< |a:o - yol + 


c + Ti 


for all n > 4. (The reader is invited to recall that uti is still ui (Ti,7o,xo) and that ui E Aq is left 
continuous). One gets, on Ti < T, 


v'Y0,^0,'^,'v _ yVo, 


cTi 


cTi 


< min A/Ixo-yol H-, 2fco + - <|xo-yo| H-h^+d/cQ- 


n 


n 


n 


n 


(13) 


^1 |2 I C (Ti + dfco + 1 ) 

< ko - yol H- 


n 


whenever n> c. Finally, using Pi) and (HID, we get 


h ( 70 , - h ( 70 , | < Lip (h) |^,o>*o,«,^;7o _ ^ 0 , 2 ^ 0 , 


+ 


n 


< Lip{h) ko - yol + 


{Lip {h) + 1) c + Lip (h) t 


n 


for all s < T A Ti . 

Step 2. We continue the construction on [Ti,T 2 ). By abuse of notation, we let U 2 {s) := 
U 2 for s > 0 and similar for V 2 - We set 

:= wllo<s<Ti + W ^jy 0 ’^ 0 ,u,v ^ j^^o,xo,u,v ^yyo,w ^ (Q) ^ (Q)^ l^^y^ . 

It is clear that this control process is predictable. We apply the same method as in Step 1 (uj- 
wise) on the (stochastic) time interval [Ti, (Ti + A r 2 ] , then on [ri,(ri + f) ATa], etc. We 
construct a sequence of control processes and, by choosing m large enough, we establish 

the existence of some w'^ such that 
(14) 


j^'yo,xo,u,v _ 
y70,^0,u,v _ Yyo,w‘'^ 

^T2 ^T2 


Y70,xo,u,v _ yPO,' 

^T2 ^T2 


< 

< 


Y'yo,xo,u,v _ •Yyo,w 
y 70 ,xo,u,v _ 

^T2- ^T2- 


+ ^ < |l'o - 


h{l, 


< Lip (h) 


, 70 ,X 0 ,u,i; j^'yo.XQ.u^v 




< min V 2ko^ + 7 < l^o - yol + 

< ko - yol' + + # + 4 A:o^ < ko - yol' + 


’ t 


X. 


'f 0 ,X 0 ,U,V 


- yk°’ 


+ f < Lip{h) ko - yol + 


Ut,Wt 

{Lip{h)+l)c+Lip{h){t+4ko+l) 

77 ’ 


20 








































for all Ti < t < r 2 A T , P—a.s. if n > max (4, c) . We continue our construction on [0, AT], 
[0, r 4 A r] and so on to finally get a predictable process re™ such that 


(15) 


j^'yo,xo,u,v _ yj/o,' 


y10,^0,u,v 


_ vyo,w" 


< |a:o - yo\ + 


2 , c[t+(i-l)(4fco+l)] 


< min ( Vko - yof + ) + 


< ko - yo\ + 




^,o,^o,u,v_yyo,^- 


2 I c[Ti-\-i{4ikQ-\-l)] 




< Lip (h) |xo - yol + 


for all i < m, Tj_i < t < Ti AT, P—a.s. 

Let us note that in the same way as [ISl Inequality 3.27], one has 


E 


^-ST„, 


< l-(5 


f 


g-t(<5+Amax)^^ ] ^ 


A. 


(5 + Aj] 


where Amax = sup |A.y (x,tt,u)| . Then, using the estimates (flbll . one gets 

(7,3;,u,i;)sMx xUxV 


E 


•f 






•jr) dt 


< E 


E™ o' (lip (h) |xo - yol + (^»pW+l)c+L^/^)[t+h4fco+l)] ^ 

+ 2/lmaxlT^<r 5e~^^dt + 2hma.^e~^'^ 

r /' u\ roo 

5e~^^tdt + 2hraaxe~^'^ 


(Lip(/i) + l)c + mLzp(h) (4A:o + 1) Lip{h) 

< Lip [h] \xq — yo H- 7 =-1-T=— 

Vn Vn 

e-^Trn _ ^-ST' 


+ 2/lmaxlE 

< Lip{h) |xo - yol + 


'^Trr,<T 

Lip (h) + 1 


n 


C + m (4/co + 1) + ^ ) + 2/iniaxe + 2/lmaxIE 


^-STrr. 


< Lip {h) \xo - yol + ^ (c + m {Ako + 1) + ^ ) + 2/imaxe + 2K 


(5 + Aj] 


The proof of our Lemma is now complete by picking T = m = ni. 


7 Appendix 

We provide, in this appendix, the key elements of proof leading to Theorem 1 131 As we have already 
hinted before, the proof relies on the same arguments as those developed in [40] combined with 
dynamic programming principles developed in |48] . 

If one assumes that K <ZB (0, /cq) is convex and invariant w.r.t. the PDMP governed by (/, A, Q), 
then one modihes the dynamics such that for p E {/, A}) = 0, if x ^ i?(0,A:o + l), 

{x, u) = p.y (x, n), if X E K and setting, for example, (0, x, u) = IIk (x) — x + py (0, IIk (x) , u), 
for all X E M^. Here, Hk stands for the projector onto IK. In this way, all jumps x i-A x + 
y (0, X, u) = Hk (x) + g (0, Hk (x) , u) take the trajectory in K (by invariance of this set) and, if the 
trajectory reaches B (0, /cq + 1), it stays constant. For the extended dynamics (constructed from 
this modification as in Subsection El]), one gets 

fy (x, u^,u^) = fy (^x + u‘^,u^) = 0, for all x E such that |x| > ko + 2, 

X Tpy [0, X, u^,u^) = [x + + gy (0, X + n^)] — E K—C B (0, ko + 1), for all x E M'^, 
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for all G U and all \v?‘\ < 1. 

It follows that the set := B (0, /cq + 2) is invariant w.r.t the extended dynamics. In fact all 
sets B (0, /cq + n), n >2 are invariant. Let us emphasize that this construction is the only point in 
which the convexity of IK plays a part and it can be avoided by further assumptions. 

Let us fix, for the time being, d > 0 and n > 1. 


7.1 Dynamic Programming Principle(s) for (Time) Piecewise Constant Policies 

The first ingredient is to provide dynamic programming principles and uniform continuity for the 
value functions given with respect to piecewise constant policies with respect to the initial and 
auxiliary systems (cf. Subsection 16.11) . In addition to Aq = .Aq (U) and A'^^ := (U), one 

introduces Bq = Aq (UxB (0,1)) and 13”^ = A^^ (VxB (0,1)) and 


Ty, x) = inf (5E 


e-^^h I r 






'y,x,u^ ,eu^ 


+ eu7,u} ) dt 


for all initial data 7 G M, x G i? (0, /cq + 3). 

One begins with proving the dynamic programming principles. 

( 7 , x) = inf E 6e-^^h ut) dt + 




Ue ( 7 ,x) = inf E 


and 


rTiAT j —Sti. f-n'y,x,u^,su^ , 2 1 

Jo ,x/' ’ ’ +eut,ul 


+g-5(TiAT)^5,n 


dt 


\x: 


TiAT 


The arguments are similar to those employed in [AH]. We will only emphasize what changes when 
using controls from A^^ (or B^^) instead of the (more) classical Aad- 
Following [38], we introduce 


w 


M.n 


( 7 , x) := inf (-y^ X, u) 


UGA, 


where 

:= E 




whenever M > 1. The initial value w^’’^ is given with respect to the deterministic control problem 
(with no jump) and it is standard to check that it is Holder continuous (the Holder exponent may 
be chosen where Lip{f) is the Lipschitz constant for for all 7 G M and the Holder 

constant only depends on the Lipschitz constants and supremum norm of / and h). In particular, 
the continuity modulus of (resp. We’"' defined w.r.t. Bq) is independent of n (resp. n and e). 

Step 1. If G BUC (MxR^;M) , then the dynamic programming principle holds true for 


= inf E 


wM,n x) G M X T & n : 

,5 e-^^h Ut) dt + ( 7 , It,>t 


The proof is identical with the proof of |48[ Lemma 3.1]. The reader needs only note that the 
control policy given by [351 Eq. (3.5)] of the form 

u {0, y, t) := u {9, y, t) 1[o,t] {t) + u* {o, t-T) lt>T 

belongs to Aq if u and u* belong to ^q. 
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Step 2. Since B (0, fco + 3) is invariant with respect to the extended PDMP, one has E 

BUC (Mxi? (0, feo + 3)) and, for every a > 0, there exists a a—optimal control policy u* E Aq 
such that 

( 7 , x,u*)< i-f, x) + a, 

for all X E -B (0, ko + 3). 

Again, the proof is identical with the proof of the analogous Lemma 3.3 in [38] and based on 
recurrence. The reader needs only note that, for r > 0, there exists a finite family {x^ ■ k = l,m} 
such that 

_ m 

B(0,/co+3)c UB(xfc,r). 

k=l 

Then, the control policy u defined after (3.18) in |l 8 | belongs to if Uk belong to ^g, for all 
k = l,m. We also wish to point out that the estimates leading to the continuity modulus of w^’’^ 
only depend on the Lipschitz constants and the supremum of h, /, g and A but are independent 
of the control policies. In particular, this allows one to work with a common continuity modulus 
^(5,M £qj. all n > 1 and e > 0 . 

One concludes using the same arguments (no particular changes needed) as those in [381 Theo¬ 
rem 3.4]. Due to [38l Inequality 3.27], one gets 


sup 

(7,3;)SMxi?{0,fco+3) 


{-f,x)-w^’^ (7,x) 


< ca 


M 


where c > 0 and 0 < a < 1 are independent of n (c can be chosen as in [38| equal to 2/max and 
a as in |38l Page 1120, last line] to be 1 — J < 1). The same is true for 

nf’"" — w^’"' for e > 0. In particular, 


nf’- ( 7 , x) - nf’" ( 7 , y) < (k - y\) + 2c 


5 + An 


M 


i.e. the continuity modulus of nf’” can also be chosen independent of n > 1 and e > 0 (we identify 
Vq with This common continuity modulus will be denoted by , i.e. 


(16) (r) = sup sup 

n>l,e>0 7SM 
\x-y\<r 


nf’"(7,x)-nf’"(7,x) 


r > 0 , u)° ( 0 ) := lim io° (r) = 0 . 
r^O 
r >0 


The reader will note that (r) > cr, for some c > 0, where the equality corresponds to the 
Lipschitz case. 


7.2 Estimates and Proof of Theorem 1131 

We begin with the following convergence result. 


Proposition 17 For every 5 > 0 there exists a decreasing function : M+ —> M+ that satisfies 
limg^o i^) = 0 o-nd such that 


(17) 


sup 

xeK+ 


- V 


S,n , 


( 7 , x) < (e), 


for all n > 1 and all e > 0 . 

Proof. The proof is similar to the one of [331 Theorem 3.6]. However, we present the arguments 
for reader’s sake. Let us fix 7 E M, x E HC'*' and e > 0. The definition of the value functions implies 
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that x) < ( 7 , x). Standard estimates yield the existence of some positive constant C > 0 

which is independent of 7 , x, of n > 1 and e > 0 such that 


(18) 


$ 


0,x,u^ 


< Ce, 


for all t G [0,1] , and all (u^,u^) G Bq. We recall that is the unique solution of the 

deterministic equation 


0,x,w^,w^;7 _ 


$ 


0,x,u^ ,€U^;'y 


Ut^Ut 


dt=L(^ 


z0,x,u^ . 2 


dt, 




0 


= X. 


The constant C in (|18|) is generic and may change from one line to another. We emphasize that 
throughout the proof, C may be chosen independent of x G M^, n > 1, e > 0 and of {v},u^) G B^ 
(it only depends on Lipschitz constants and bounds of /, A, g and h). The dynamic programming 
principle written for yields 
(19) 


u'^’'‘(7,x) < E 



5e-^^h 


( p7,x,11^,0 Y'i,x,u^fi l\ 

^ t J 


dt + 


/-p7,a;,n'^,0 '^7,x,ul,0\ 
TiAl ’ ^TiAl J 


for all G Aq. We consider an arbitrary admissible control couple (u^,u^) G Bq. For simplicity, 
we introduce the following notations: 


u\ = vd (x,t), i = 1 , 2 , 

(t) = A.y ^ ^ j j'g) 


A^’7t) = A7 


: 0,a;,n^,en^;7 


+ eut,uA , A’ (t) = exp - / A ’ {s)ds] , 


for all t > 0. We denote the right-hand member of the inequality (jl9p by I. Then, I is explicitly 
given by 


/ = { A‘ (t)A" (t) /' Se-^’h ( 7 , ul'j dadt 

JO ^ ^ ^ ^ 

( 1 ) ^ be-^^h ( 7 , $ 0 ,-,-b 0 ; 7 ^ ( 1 ) g-75,n ^ 0 ,x,nb 0 ; 7 ^ 


+ 


= h + h + h + h- 


Using the inequality (USD, one gets 


( 20 ) 

( 21 ) 


h < (t) be-^^h ( 7 , 

Is < (1) be-^% (7,+ eulu\ 


) dt -\- Cs. 


dsdt + Ce, 
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For the term I 2 , with the notation (1161) . one has 

h 


—St I „,S,n f n 




< [ (t) 

Jo 

+ C* + Lo^ (Ce)^ 

Jo Jr^ ^ 


< 

( 22 ) 

+ 




0,x,u^ ,£u^]‘y 


+ 57 + {'y,ul,de) dt 


+ 57 [d, Q° ( 7 , d^) dt 


f A^’^(t)A^’^ (t) e sup ^'^’"■( 2 ) — r;f’”'(z) + C ("e + (C'e)') 

Jo ) eeM,^eK+ ' 

Finally, 

U < A ^’2 (1) ^-S^S,n ^0,x,«h£«2.7^ +c{e + J (Cs)) 


(23) 


< A^’2(i)g-<5^5,n / ^ ^0 ,x,«1 ,£«2;7\ ^ ^^’2^1)6"^ SUp V^’^{ 0 , z) - z) + C (cU^ (Cs) + ff) . 

^ 0eM,2eK+ ^ 


We substitute (I2U|1 - (I^5|1 in (flUjl . We take the infimum over the family of (n^,tt^) G Bq and use the 
dynamic programming principle to have 


( 7 , x) < t;f’”( 7 , x) + C (e + 


+ ( / A^’^(t)A^’^ (t) e '^*dt + A^’^(l)e J sup 

0eM,2GK+ 


v^’-(z}-v^,’-(z} 


We notice that 
/■i 


'0 


A^’^(t)A^’^ (t) e-^^dt + A^’^{l)e-^ = 1 - d / e 

A 


1. 


1 .,,2 


,ul,eu'i ds 


e-^^dt 


^ -max ^ 


Q — ('^max + f^) 


Amax A d Amax + d 


Thus, 

( 7 , x) - xf’”( 7 , x) < C (e + cj 

+ 


+ 


^max + d '^max H” ^ 


e I sup 

0eM,zeK^ 


v^^^ie,z)-vp^ie,z) 


,S,nj 


Here, Amax := sup {A..^ (x,n) : 7 G M, x G M'^,n G U} < 00. The conclusion follows by taking the 
supremum over d G M and x G K"*" and recalling that C is independent of x and e > 0 (and n > 1). 

■ 

We consider [p^) a sequence of standard mollifiers i.e. (y) = jwyO (f) , y G M-^, e > 0, where 
p G C°° (R-^) is a positive function such that 

Supp{p) C B (0,1) and / p{x)dx = 1. 


We introduce the convoluted functions 

5,71 


( 7,0 := Vs'^ {'!,■) *Pe- 

In analogy to HOI Lemma 3.5], one gets 
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Proposition 18 The value functions are such that 


sup 

xGK 


sup 

x,yGK., y^x 

sup 

xEK 


v(e) (7, x) + (7, x) ) < C^e ^ (e + (e)) , 

d 5 ,n / \ o < 5 ,n / \ 

' ^ |i-/ ’ ^ ^ ^ (1^ - y\) 

(7, x) - (7, x) < io^ (e) + 7 ]^ (e), 

for all 7 G M. Here, is a positive constant independent of £ > 0 , n > 1 and 7 G 


Proof. To prove the first inequality, one recalls the definition of uf’?. Then, due to Proposition fT71 


and using the notation (flUjl , one gets 


P)' 


dxixl^^ (.x) 


£ ^ I xf’” (x - ey) (y) dy 


e ^ [ (xf’” (x - ey) - (x)) d,,ip (y) dy 


dB(0,l) 


J h J 

J/6B(0,1) 



< C^£-^u}^ (e). 


Similarly, 


d S.n / \ o S.n ( \ 
(x) - (y) 


.-1 


[_ (ve’'' {x - £z) - uf’" (y - ez)) d^ip {z) dy 


< CW(|x-y|) 


Moreover, again with the notation (| 16 h and the help of Proposition fT71 one gets 


sup 


5,n / N 
X(^) (x) - V 


s.n 


(x) 


= J [xe'' {x - ey) - (x) + uf’*" (x) - (x)) p (y) dy 

yeB{0,l) 

< uj^ (e) + (e). 


The proof of our proposition is now complete. ■ 

We now come to the proof of the main convergence result. 

Proof, (of Theorem I13p . Let us fix (m^,u^) G U x i? (0,1) .The dynamic programming principle 
written for uf’” yields 


xf’” (7, x) < E 


Jo 


-he 


Se-^*h ^P 

—S(^TiAn~ 






■y,x,u^,eu^ 


+ £Uf,u} ) dt 


) < 5,1 
IVe’ 


-p"f ,x ,u^ ,eu‘^ Y'yx,u^ ,evP‘ 
'•'TiAn-l ’"^TiAn-l 


where G Bq and x G K. In particular, if = (n, y) G U x i? ( 0 , 1 ), for t G [ 0 ,re ^), 

one notices that on [O, T\ A , 


\r'y,x—ey,u^,eu‘^ fl 


ey and P 


•y,x,u^,eu 

t 


2 


= r 


■y,x,u^ ,0 
t 


As consequence, the (law of the) first jump time starting from (7, x — ey) when the trajectory is 
controlled by the couple (n^,eu^) given above only depends on u and x (but not on e, nor on y). 

To emphasize this dependence, we denote it by Similarly, ^ has the 

same law as — £yj- On gets 


xf’"' (7, X - ey) < E 


+e 


rriCL 

r 1 

Jo 

-S{T\ 


An- 




dt 


iX,U 


An 


) < 5 ,”. 

>Vs 


p 7 ,x,u \0 ^ 7 ,x,ui ,0 _ 

-I- rr^x.u * .._1 5 ^rpX,U^^_l Cy 


An- 


26 




























It follows that 


(7,®) 


< E 


f.Tf’“An- 


Se-^^h w7,a:,«,0 l\ ,, , -5(rf’“An-l) <5,n /-p7,a;,n,0 -^7,a:,n,0 \ 

oe n\l^ ,Aj ,U^jat + e ^(e) Tf’“An-l’^rf’“An-V 


Applying Ito’s formula to ^p7:3;,n,0^ ^7,3;,u,o^ [0,T*’“ A n and recalling that = rt prior 

to n~^, it follows that 

A 7 ’"’“’°) 


0 < E 


r-rf’“An-i 


—St 


dt 


= E 
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5 ,n 
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(5 (|/i|i + C"^e ^ (e + (e))) 1/71 + l/l^ tC"^e ^(e + a;'^(e)) 

+ |/liCW(^)+ 2 |/i 7 |A 7|/|7 


+ l^li ^ (e + (e)) (1 + 157) 1/7 t 

< E [r7’“ A n~^] [ s(h ( 7 , X, w) - ( 7 , x)) + ( 7 , x) ^ 


dt 


+ E [r7’“ A n"^] 




+ e w 


n 


n 


The generic constant is independent of x G IK, 7 G M, x G U, e > 0 and n > 1 and may change 
from one line to another. As consequence, 


5 (/i ( 7 , X, u) - ( 7 , x) j + T^x";^ ( 7 , x) > -C 


(e) 1 5 7177 


+ e tx 


n 


n 


We fix (for the time being), the initial configuration ( 70 , xq) G M x IK and an arbitrary control 
G Aad- We apply the previous inequality for ( 7 ,x) = ^ integrate the 

inequality with respect to e~^^dt on [0,T] for T > 0 and use Ito’s formula to get 


E 


6e-^% A7°’"°’“'’°, n^) dt > ( 70 , xq) - ('r7o,xo,«bo^ ^ 70 . 

1 (^) , - 1 , ,5 f\f\i 


xo,tt^,0 


-C^ 
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+ e u 
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One lets T —)• 00 and takes the infimum over G Aad to get 

'i-+ (s) , ^- 1 , A 7177 


V ( 7 o,a^o) > ( 7 o,a:o) - 


+ e io 
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Finally, using the third estimate in Proposition 1181 one gets 

^ + , ^- 1 ,, A 7177 


( 70 , a:o) > v^’"- ( 70 , xo) - C 


+ S OJ 


n 


n 


+ J{e) + r^^{e). 


The conclusion follows by taking e = ^cx^ (^^)) ’ some 1 > 5 > 0 (e.g. rj = ^). Our result 

is now complete. ■ 
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